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Program Description 1 

First year experience courses for students in developmental education 
are designed to ease the transition to college for the large numbers 
of students in need of developmental (or remedial) education. These 
courses were developed because approximately 40% of first- or 
second-year undergraduates in 2-year colleges and almost 30% of 
first- or second-year undergraduates at 4-year colleges report taking 
developmental courses; these numbers are higher for first genera- 
tion and minority students. 2 First year experience courses (also called 
success courses, study skills, student development, or new student 
orientation courses) 3 occur during the first year of students’ enroll- 
ment in postsecondary education. The aim of these courses is to sup- 
port the academic performance, social development, persistence, and 
degree completion of postsecondary students with developmental 
needs. 4 Although first year experience courses vary in terms of con- 
tent and focus, most are designed to introduce students to campus 
resources, provide training in time management and study skills, and 
address student development issues. For students in developmental 
courses, the courses are often linked with or taken concurrently with 
developmental courses. 5 
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This intervention report presents 
findings from a systematic review 
of first year experience courses for 
students in developmental education 
conducted using the WWC Procedures 
and Standards Handbook, version 
3.0, and Developmental Students 
in Postsecondary Education review 
protocol, version 3.0. 


Research 6 

The What Works Clearinghouse (WWC) identified one study of first year experience courses for students in developmental 
education that both falls within the scope of the Interventions for Developmental Students in Postsecondary Educa- 
tion topic area and meets WWC group design standards. This one study meets WWC group design standards without 
reservations. This study included 91 1 freshman college students in developmental education enrolled at one technical 
community college in the United States. 

The WWC considers the extent of evidence for first year experience courses for students in developmental educa- 
tion to be small for three student-level outcome domains— academic achievement, progress through develop- 
mental education, and credit accumulation and persistence. There were no studies that meet WWC group design 
standards in the three other domains specified as eligible in the review protocol, so this intervention report does 
not report on the effectiveness of first year experience courses for students in developmental education for those 
domains. (See the Effectiveness Summary on p. 5 for more details of effectiveness by domain.) 
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Effectiveness 


Based on the one study that meets WWC group design standards, first year experience courses for students in 
developmental education were found to have no discernible effects on academic achievement, progress through 
developmental education, and credit accumulation and persistence for postsecondary students. 


Table 1. Summary of findings 7 




Improvement index (percentile points) 




Outcome domain 

Rating of 
effectiveness 

Average 

Range 

Number of 
studies 

Number of 
students 

Extent of 
evidence 

Academic achievement 

No discernible effects 

+1 

na 

1 

911 

Small 

Progress through 
developmental education 

No discernible effects 

-1 

na 

1 

911 

Small 

Credit accumulation 
and persistence 

No discernible effects 

-3 

na 

1 

911 

Small 


na = not applicable 
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Program Information 

Background 

Large numbers of students arrive at college unprepared for college-level coursework. They also lack the strong 
study skills and coping strategies required to effectively navigate the learning and social environments in higher 
education. First year experience courses for students in developmental education are designed to help under- 
prepared postsecondary students transition to college, understand institutional expectations, and make effective 
academic and career decisions. 8 Barefoot and Fidler (1992) 9 identified five types of first year experience courses: 
extended orientation, academic seminars with uniform or variable content, introduction to a discipline or profes- 
sional seminars, and basic study skills seminars. First year experience courses for students in developmental 
education tend to emphasize study skills more than first year experience courses for general education students 
and are often taken concurrently with developmental requirements. As reported by the National Resource Center 
for the First- Year Experience and Students in Transition, first year experience courses are required for all first-year 
students at approximately 52% of 4-year institutions in the United States. Flowever, when they are not required for 
all students, they are frequently required for special populations (e.g., academically underprepared students, stu- 
dents in specific majors, and students taking developmental courses). 10 

Program details 

The one study of the first year experience courses for students in developmental education that met WWC group 
design standards reviewed in this intervention report examined the basic study skills approach (Rutschow, Cul- 
linan, & Welbeck, 2012). The course was redesigned from an existing study skills course offered at the college. The 
course focused on helping students take responsibility for their learning and included content on academic skill- 
building via class presentations, journal writing, quizzes, and an end-of-semester course project. The course was 
offered to students in need of one or more developmental courses. 


Cost 

Rutschow, Cullinan, and Welbeck (2012) do not report on the costs of the first year experience course for students 
in developmental education program included in the one study reviewed in this intervention report. 
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Research Summary 

The WWC identified 19 eligible studies that investigated the effects of Table 2. Scope of reviewed research 
first year experience courses for students in developmental education 
for college freshmen. An additional 31 studies were identified but do 
not meet WWC eligibility criteria for review in this topic area. Citations 
for all 50 studies are in the References section, which begins on p. 7. 

The WWC reviewed 19 eligible studies against group design standards. 

One study is a randomized controlled trial that meets WWC group design standards without reservations. This 
study is summarized in this report. The remaining 18 studies do not meet WWC group design standards. 

Summary of studies meeting WWC group design standards without reservations 

Rutschow, Cullinan, and Welbeck (2012) conducted a randomized controlled trial to evaluate a first year experience 
course for students in developmental education at a technical community college in the southeast United States. 
The intervention group included 458 students who were at least 18 years of age, were required to take one or 
more developmental education courses, were new or continuing students with less than 20 credit hours, were not 
majoring in a degree that required the success course, had not previously participated in the success course, and 
were willing to participate. The evaluation took place over three semesters (spring 2008 through spring 2009). The 
two-credit course was designed to assist students in overcoming both academic and personal challenges that 
might interfere with college success. The curriculum had eight core principles: 1) Accepting personal responsibility, 
2) Self-motivation, 3) Self-management, 4) Interdependence, 5) Self-awareness, 6) Lifelong learning, 7) Emotional 
intelligence, and 8) Belief in self. The course was typically taught by two instructors. Classes were generally collab- 
orative in nature, with group discussions being more common than lectures. Students were often asked to critically 
reflect on their personal experiences and write journal entries. In addition to socioemotional skill development, 
the students received study skills instruction, such as note taking, time management, and test preparation. Addi- 
tional class requirements included traditional classroom assignments such as a term paper, a class presentation, 
and tests. Students in the comparison group had access to the college’s typical student support services, which 
includes advising, tutoring, career counseling, and assistance in transferring to a 4-year college. 

Summary of studies meeting WWC group design standards with reservations 

No studies of first year experience courses for students in developmental education met WWC group design 
standards with reservations. 


Grade 

Postsecondary 

Delivery method 

Whole class 

Program type 

Practice 
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Effectiveness Summary 

The WWC review of first year experience courses for students in developmental education for the Interventions for 
Developmental Students in Postsecondary Education topic area includes postsecondary outcomes in six domains: 
academic achievement, progress through developmental education, credit accumulation and persistence, access 
and enrollment, attainment, and labor market. The one study of first year experience courses for students in 
developmental education that meets WWC group design standards reported findings in three of the six domains: 

(a) academic achievement, (b) progress through developmental education, and (c) credit accumulation and persis- 
tence. The findings below present the authors’ estimates and WWC-calculated estimates of the size and statistical 
significance of the effects of first year experience courses for students in developmental education on freshman 
college students. For a more detailed description of the rating of effectiveness and extent of evidence criteria, see 
the WWC Rating Criteria on p. 18. 

Summary of effectiveness for the academic achievement domain 

One study that meets WWC group design standards without reservations reported findings in the academic 
achievement domain. 

Rutschow et al. (2012) reported on the percentage of students who passed all courses during the program semes- 
ter and the percentage of students who received a grade point average (GPA) of “C” or better in nondevelopmental 
courses three semesters after the program. The authors reported, and the WWC confirmed, that there were no 
statistically significant differences between first year experience course participants and comparison participants on 
either the percentage of students passing all courses or the percentage of students receiving a GPA of “C” or bet- 
ter. The WWC characterizes this finding as an indeterminate effect. The mean effect reported is neither statistically 
significant nor substantively important. 

Thus, for the academic achievement domain, one study showed neither a statistically significant nor substantively 
important effect. This results in a rating of no discernible effects, with a small extent of evidence. 


Table 3. Rating of effectiveness and extent of evidence for the academic achievement domain 


Rating of effectiveness 

Criteria met 

No discernible effects 

None of the studies show 
statistically significant or 
substantively important effects, 
either positive or negative. 

In the one study that reported findings, the estimated impact of the intervention on outcomes in the academic 
achievement domain was neither statistically significant nor large enough to be substantively important. 

Extent of evidence 

Criteria met 

Small 

One study that included 911 students in one technical community college reported evidence of effectiveness in 
the academic achievement domain. 


Summary of effectiveness for the progress through developmental education domain 

One study that meets WWC group design standards without reservations reported findings in the progress through 
developmental education domain. 

Rutschow et al. (2012) reported on the percentage of students who passed their developmental math, reading, and 
English courses (separately). The authors reported, and the WWC confirmed, that there were no statistically signifi- 
cant differences between first year experience course participants and comparison participants on the percentage 
of students who passed their developmental math, reading, or English courses. The WWC characterizes the mean 
effect across the two measures in this domain as an indeterminate effect. 
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Thus, for the progress through developmental education domain, one study showed neither a statistically significant 
nor substantively important effect. This results in a rating of no discernible effects, with a small extent of evidence. 


Table 4. Rating of effectiveness and extent of evidence for the progress through developmental 


education domain 

Rating of effectiveness 

Criteria met 

No discernible effects 

None of the studies show 
statistically significant or 
substantively important effects, 
either positive or negative. 

In the one study that reported findings, the estimated impact of the intervention on outcomes in the progress 
through developmental education domain was neither statistically significant nor large enough to be substantively 
important. 

Extent of evidence 

Criteria met 

Small 

One study that included 911 students in one technical community college reported evidence of effectiveness in 
the progress through developmental education domain. 


Summary of effectiveness for the credit accumulation and persistence domain 

One study that meets WWC group design standards without reservations reported findings in the credit accumula- 
tion and persistence domain. 

Rutschow et al. (2012) reported on the percentage of students who registered for any courses and the number of 
credits earned following the third semester after the program. The authors reported, and the WWC confirmed, that 
there were no statistically significant differences in credit accumulation and persistence between students who 
participated in the first year experience course and those who did not. The WWC characterizes this finding as an 
indeterminate effect. The mean effect reported is neither statistically significant nor substantively important. 

Thus, for the credit accumulation and persistence domain, one study that met WWC group design standards with- 
out reservations showed neither a statistically significant nor substantively important effect. This results in a rating 
of no discernible effects, with a small extent of evidence. 


Table 5. Rating of effectiveness and extent of evidence for the credit accumulation and persistence domain 


Rating of effectiveness 

Criteria met 

No discernible effects 

None of the studies show 
statistically significant or 
substantively important effects, 
either positive or negative. 

In the one study that reported findings, the estimated impact of the intervention on outcomes in the credit 
accumulation and persistence domain was neither statistically significant nor large enough to be substantively 
important. 

Extent of evidence 

Criteria met 

Small 

One study that included 911 students in one technical community college reported evidence of effectiveness in 
the credit accumulation and persistence domain. 
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Appendix A: Research details for Rutschow et al. (2012) 

Rutschow, E. Z., Cullinan, D., & Welbeck, R. (2012). Keeping students on course: An impact study of a 
student success course at Guilford Technical Community College. New York: MDRC. 

Table A. Summary of findings Meets WWC group design standards without reservations 


Study findings 
Average improvement index 


Outcome domain 

Sample size 

(percentile points) 

Statistically significant 

Academic achievement 

911 students 

+1 

No 

Progress through 
developmental education 

911 students 

-1 

No 

Credit accumulation 
and persistence 

911 students 

-3 

No 


Setting The study took place at Guilford Technical Community College located in Guilford County, 
North Carolina. The college is the only community college in the county. In 2008, the total 
enrollment at the college was 1 1 ,226. The college is divided across three cities. The first year 
experience course was offered at three of Guilford’s campuses. The first year experience 
course was administered across three semesters to three semester-based cohorts. Multiple 
sections of the course were offered, with 24 instructors teaching the course in the first semes- 
ter. Over the 3 years of the intervention, 33 success course sections were offered. Course 
sections were typically taught by two instructors. 

Study sample All 91 1 students were required to take at least one developmental course. Female students 
represented 69% of the intervention group and 68% of the comparison group. The mean age 
of the sample was 25 for both groups. The ethnic breakdown for both groups was Black (58% 
intervention, 59% comparison); White (28% intervention, 29% comparison); Hispanic (6% 
intervention, 6% comparison); and Other (8% intervention, 7% comparison). Of the sample, 
24% of the intervention group and 28% of the comparison group were the first members of 
their families to attend college. 


Intervention The first year experience course was a redesigned version of an existing study skills course 

group and was modeled on Skip Downing’s On Course: Strategies for Creating Success in College 

and Life curriculum. The two-credit course was designed to assist students in overcoming 
both academic and personal challenges that might interfere with college success. The cur- 
riculum had eight core principles: 1) Accepting personal responsibility, 2) Self-motivation, 3) 
Self-management, 4) Interdependence, 5) Self-awareness, 6) Lifelong learning, 7) Emotional 
intelligence, and 8) Belief in self. The course was typically taught by two instructors. Classes 
were generally collaborative in nature, with group discussions being more common than lec- 
tures. Students were often asked to critically reflect on their personal experiences and write 
journal entries. In addition to socioemotional skill development, the students received study 
skills instruction, such as note taking, time management, and test preparation. Additional 
class requirements included traditional classroom assignments such as a term paper, a class 
presentation, and tests. 
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Comparison 

group 


Outcomes and 
measurement 


Support for 
implementation 


This study utilized a “services as usual” comparison condition. Students in the comparison 
group (as well as the intervention group) were eligible to participate in the college’s other stu- 
dent support services, which included advising, tutoring, career counseling, and assistance in 
transferring to a 4-year college. 

The authors reported several eligible outcomes, all of which were obtained from official tran- 
scripts. The eligible primary outcomes were: 1) percent of students who registered for any 
courses (credit accumulation and persistence domain); 2) number of regular credits earned 
(credit accumulation and persistence domain); 3) percent of students who passed develop- 
mental math (progress through developmental education domain); 4) percent of students who 
passed developmental English (progress through developmental education domain); 5) percent 
of students who passed developmental reading (progress through developmental education 
domain); 6) percent of students who passed all courses (academic achievement domain); and 
7) percent of students who received a grade point average (GPA) of “C” or better in nonde- 
velopmental courses (academic achievement domain). All eligible outcomes were measured 
at each of three time points (program semester, first postprogram semester, and cumulative 
from program to third postprogram semester) except for “percent of students who passed all 
courses”, which was only measured at the first time point. This review was created using data 
from the last follow-up period available, as described in the review protocol. 

Staff who taught the success course during the first semester had all previously been trained 
on the On Course curriculum. In addition to the 3-day seminar, some staff also attended a 
week-long seminar which included material on how to train others on the curriculum. In addi- 
tion to teaching, these individuals recruited and trained additional instructors who taught 
sections of the course in the final two semesters. Some faculty members received additional 
professional development by attending conferences or workshops. During the first semester 
of the intervention, all instructors met every month to discuss the course and best practices 
for teaching. After the first semester, however, only one meeting was held at the beginning of 
the semester, and instructors were asked to schedule one-on-one meetings in lieu of monthly 
group meetings. 
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Appendix B: Outcome measures for each domain 


Academic achievement 

Percentage of students who passed 
all courses 

Academic achievement was assessed as the percentage of students who passed all of their courses during the 
program semester. 

Percentage of students who 
earned a GPA of “C” or better In 
nondevelopmental courses 

Academic achievement was measured as the percentage of students who received a GPA of “C” or better in 
nondevelopmental courses following the third postprogram semester. This outcome is referred to as “Term GPA” 
in the study text and is coded by the authors as a categorical variable: 3.0 to 4.0, 2.0 to 2.9, 1 .0 to 1 .9, 0 to 
0.9, and No GPA. The WWC recoded this into a binary variable that compares students with a “C” or better GPA 
to those with less than a “C” average. Students with no GPA were not included in this computation. 

Progress through 
developmental education 

Passed developmental math 
course 

Progress through developmental education was measured as the percentage of students who had passed their 
developmental math course after the third postprogram semester. 

Passed developmental reading 
course 

Progress through developmental education was measured as the percentage of students who had passed their 
developmental reading course after the third postprogram semester. 

Passed developmental English 
course 

Progress through developmental education was measured as the percentage of students who had passed their 
developmental English course after the third postprogram semester. 

Credit accumulation and persistence 

Percentage of students registered for 
any course 

Credit accumulation and persistence was measured as the percentage of students who were registered for any 
course after the third postprogram semester. 

Number of credits earned (regular, 
college-level courses) 

Credit accumulation and persistence was assessed as the average number of regular, college-level credits 
earned after the third postprogram semester. 
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Appendix C.1: Findings included in the rating for the academic achievement domain 


Mean 

(standard deviation) WWC calculations 


Outcome measure 

Study 

sample 

Sample 

size 

Intervention 

group 

Comparison 

group 

Mean 

difference 

Effect 

size 

Improvement 

index 

p-value 

Rutschow etal., 201 2 a 

Percentage of students who 
passed all courses during the 
program semester 

College 

students 

911 

32% 

36% 

-4% 

-0.11 

-4 

>.10 

Percentage of students who 

College 

667 

56% 

57% 

-1% 

-0.03 

-1 

.67 


earned a GPA of “C” or better students 
in nondevelopmental courses 

Domain average for academic achievement (Rutschow et al., 2012) -0.07 -3 Not 

statistically 

significant 


Domain average for academic achievement across all studies -0.07 -3 na 


Table Notes: For mean difference, effect size, and improvement index values reported in the table, a positive number favors the intervention group and a negative number favors 
the comparison group. The effect size is a standardized measure of the effect of an intervention on outcomes, representing the average change expected for all individuals who are 
given the intervention (measured in standard deviations of the outcome measure). The improvement index is an alternate presentation of the effect size, reflecting the change in an 
average individual's percentile rank that can be expected if the individual is given the intervention. Some statistics may not sum as expected due to rounding, na = not applicable. 

a For Rutschow et al. (201 2), a correction for multiple comparisons was needed but did not affect whether any of the contrasts were found to be statistically significant. To compute 
the effect size for GPAs reported in the table, the WWC used the data provided in Table 4.4 of the study report. Students with no GPA were not included in the computation. The 
outcome was then dichotomized into students who earned a “C” or better GPA and those who earned less than a “C” average; the effect size was computed using the standard proce- 
dures for binary outcomes described in the WWC Procedures and Standards Flandbook. The p-value presented for the percentage of students passing all courses during the program 
semester was reported in the original study; the p-value for the “C" or better outcome was computed by the WWC. This study is characterized as having indeterminate effects because 
the reported effect size for all measures within the domain is neither statistically significant nor substantively important, accounting for multiple comparisons. For more information, 
please refer to the WWC Procedures and Standards Handbook (version 3.0), p. 26. 
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Appendix C.2: Findings included in the rating for the progress through developmental education domain 


Mean 

(standard deviation) WWC calculations 


Outcome measure 

Study 

sample 

Sample 

size 

Intervention 

group 

Comparison 

group 

Mean 

difference 

Effect 

size 

Improvement 

index 

p-value 

Rutschow etal., 201 2 a 

Passed developmental math 
course 

College 

students 

911 

49% 

48% 

+1% 

0.02 

+1 

>.10 

Passed developmental 
reading course 

College 

students 

911 

29% 

31% 

-1% 

-0.04 

-1 

>.10 

Passed developmental 
English course 

College 

students 

911 

31% 

33% 

-2% 

-0.04 

-2 

>.10 

Domain average for progress through developmental education 
(Rutschow et al., 2012) 



-0.02 

-1 

Not 

statistically 

significant 


Table Notes: For mean difference, effect size, and improvement index values reported in the table, a positive number favors the intervention group and a negative number favors 
the comparison group. The effect size is a standardized measure of the effect of an intervention on outcomes, representing the average change expected for all individuals who are 
given the intervention (measured in standard deviations of the outcome measure). The improvement index is an alternate presentation of the effect size, reflecting the change in 
an average individual’s percentile rank that can be expected if the individual is given the intervention. The WWC-computed average effect size is a simple average rounded to two 
decimal places; the average improvement index is calculated from the average effect size. The statistical significance of the study’s domain average was determined by the WWC. 
Some statistics may not sum as expected due to rounding, na = not applicable. 

“ For Rutschow et al. (2012), a correction for multiple comparisons was needed but did not affect whether any of the contrasts were found to be statistically significant. The p-values 
presented here were reported in the original study. This study is characterized as having indeterminate effects because the reported effect size for all measures within the domain is 
neither statistically significant nor substantively important, accounting for multiple comparisons. For more information, please refer to the WWC Procedures and Standards Handbook 
(version 3.0), p. 26. 


.02 -1 na 


Domain average for progress through developmental education across all studies 
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Appendix C.3: Findings included in the rating for the credit accumulation and persistence domain 


Mean 

(standard deviation) WWC calculations 


Outcome measure 

Study 

sample 

Sample 

size 

Intervention 

group 

Comparison 

group 

Mean 

difference 

Effect 

size 

Improvement 

index 

p-value 

Rutschow etal., 201 2 a 

Percentage of students 
registered for any course 

College 

students 

911 

91% 

90% 

+1% 

0.02 

+1 

>.10 

Number of credits earned 
(regular, college-level 

College 

students 

911 

12.8 

(na) 

12.7 

(na) 

+0.1% 

0.01 

0 

>.10 


courses) 

Domain average for credit accumulation and persistence (Rutschowetal., 2012) 0.01 +1 Not 

statistically 

significant 


Domain average for credit accumulation and persistence across all studies 0.01 +1 na 


Table Notes: For mean difference, effect size, and improvement index values reported in the table, a positive number favors the intervention group and a negative number favors 
the comparison group. The effect size is a standardized measure of the effect of an intervention on outcomes, representing the average change expected for all individuals who are 
given the intervention (measured in standard deviations of the outcome measure). The improvement index is an alternate presentation of the effect size, reflecting the change in 
an average individual’s percentile rank that can be expected if the individual is given the intervention. The WWC-computed average effect size is a simple average rounded to two 
decimal places; the average improvement index is calculated from the average effect size. The statistical significance of the domain average was determined by the WWC. Some 
statistics may not sum as expected due to rounding, na = not applicable. 

a For Rutschow et al. (201 2), a correction for multiple comparisons was needed but did not affect whether any of the contrasts were found to be statistically significant. The p-values 
presented here were reported in the original study. This study is characterized as having indeterminate effects because the reported effect size for all measures within the domain is 
neither statistically significant nor substantively important, accounting for multiple comparisons. For more information, please refer to the WWC Procedures and Standards Handbook 
(version 3.0), p. 26. 


First Year Experience Courses for Students in Developmental Education February 201 6 


Page 16 






WWC Intervention Report 


Endnotes 

1 The descriptive information for this program was obtained from Rutschow, Cullinan, and Welbeck (2012). The WWC requests devel- 
opers review the program description sections for accuracy from their perspective. Further verification of the accuracy of the descrip- 
tive information for this program is beyond the scope of this review. 

2 U.S. Department of Education, National Center for Education Statistics, 2011-12 National Postsecondary Student Aid Study 
(NPSAS:12). 

3 Barefoot, B. O., & Fidler, P. P. (1 992). The 1991 national survey of freshman seminar programming: Helping first-year college students 
climb the academic ladder (Monograph No. 1 0). Columbia: University of South Carolina, National Resource Center for the Freshman 
Year Experience. 

4 Students with developmental needs are those students who are underprepared academically to be placed in college-level courses. 
These students are identified by placement tests in reading, writing, and mathematics that are generally administered by postsecond- 
ary institutions when students enroll. Hunter, M. A., & Linder, C. W. (2005). First-year seminars. In M. L. Upcraft, J. N. Gardner, B. O. 
Barefoot, & Associates (Eds.), Challenging and supporting the first-year student: A handbook for improving the first year of college (pp. 
275-291). San Francisco: Jossey-Bass; Pascarella, E. T., & Terenzini, P. T. (2005). How college affects students. Vol. 2: A third decade 
of research. San Francisco: Jossey-Bass. 

5 Barefoot, B. O., & Fidler, P. P. (1 992). The 1991 national survey of freshman seminar programming: Helping first-year college students 
climb the academic ladder (Monograph No. 10). Columbia, SC: University of South Carolina, National Resource Center for the Fresh- 
man Year Experience. 

6 The literature search reflects documents publicly available by September 2015. The studies in this report were reviewed using the 
Standards from the WWC Procedures and Standards Handbook (version 3.0), along with those described in the Interventions for 
Developmental Students in Postsecondary Education review protocol (version 3.0). The evidence presented in this report is based on 
available research. Findings and conclusions may change as new research becomes available. 

7 For criteria used in the determination of the rating of effectiveness and extent of evidence, see the WWC Rating Criteria on p. 18. 
These improvement index numbers show the average and range of individual-level improvement indices for all findings across the 
studies. 

8 Rutschow, E. Z., Cullinan, D., & Welbeck, R. (201 2). Keeping students on course: An impact study of a student success course at 
Guildford Technical Community College. New York: MDRC. 

9 Barefoot, B. O., & Fidler, P. P. (1 992). The 1991 national survey of freshman seminar programming: Helping first-year college students 
climb the academic ladder (Monograph No. 10). Columbia, SC: University of South Carolina, National Resource Center for the Fresh- 
man Year Experience. 

10 Young, D. G., & Hopp, J. M. (201 4). 2012-2013 National survey of first-year seminars: Exploring high-impact practices in the first col- 
lege year (Research report No. 4). Columbia: University of South Carolina, National Resource Center for the First-Year Experience and 
Students in Transition. 
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WWC Rating Criteria 

Criteria used to determine the rating of a study 

Study rating 

Criteria 

Meets WWC group design 
standards without reservations 

A study that provides strong evidence for an intervention’s effectiveness, such as a well-implemented RCT. 

Meets WWC group design 

A study that provides weaker evidence for an intervention's effectiveness, such as a QED or an RCT with high 

standards with reservations 

attrition that has established equivalence of the analytic samples. 

Criteria used to determine the rating of effectiveness for an intervention 

Rating of effectiveness 

Criteria 

Positive effects 

Two or more studies show statistically significant positive effects, at least one of which met WWC group design 
standards for a strong design, AND 

No studies show statistically significant or substantively important negative effects. 

Potentially positive effects 

At least one study shows a statistically significant or substantively important positive effect, AND 

No studies show a statistically significant or substantively important negative effect AND fewer or the same number 

of studies show indeterminate effects than show statistically significant or substantively important positive effects. 

Mixed effects 

At least one study shows a statistically significant or substantively important positive effect AND at least one study 
shows a statistically significant or substantively important negative effect, but no more such studies than the number 
showing a statistically significant or substantively important positive effect, OR 

At least one study shows a statistically significant or substantively important effect AND more studies show an 
indeterminate effect than show a statistically significant or substantively important effect. 

Potentially negative effects 

One study shows a statistically significant or substantively important negative effect and no studies show 
a statistically significant or substantively important positive effect, OR 

Two or more studies show statistically significant or substantively important negative effects, at least one study 
shows a statistically significant or substantively important positive effect, and more studies show statistically 
significant or substantively important negative effects than show statistically significant or substantively important 
positive effects. 

Negative effects 

Two or more studies show statistically significant negative effects, at least one of which met WWC group design 
standards for a strong design, AND 

No studies show statistically significant or substantively important positive effects. 

No discernible effects 

None of the studies shows a statistically significant or substantively important effect, either positive or negative. 

Criteria used to determine the extent of evidence for an intervention 

Extent of evidence 

Criteria 

Medium to large 

The domain includes more than one study, AND 
The domain includes more than one school, AND 

The domain findings are based on a total sample size of at least 350 students, OR, assuming 25 students in a class, 
a total of at least 14 classrooms across studies. 

Small 

The domain includes only one study, OR 
The domain includes only one school, OR 

The domain findings are based on a total sample size of fewer than 350 students, AND, assuming 25 students 
in a class, a total of fewer than 14 classrooms across studies. 
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Glossary of Terms 

Attrition 

Clustering adjustment 
Confounding factor 

Design 
Domain 
Effect size 

Eligibility 

Equivalence 

Extent of evidence 

Improvement index 

Intervention 
Intervention report 


Multiple comparison 
adjustment 

Quasi-experimental 
design (QED) 

Randomized controlled 
trial (RCT) 

Rating of effectiveness 


Single-case design 


Attrition occurs when an outcome variable is not available for all participants initially assigned 
to the intervention and comparison groups. The WWC considers the total attrition rate and 
the difference in attrition rates across groups within a study. 

If intervention assignment is made at a cluster level and the analysis is conducted at the student 
level, the WWC will adjust the statistical significance to account for this mismatch, if necessary. 

A confounding factor is a component of a study that is completely aligned with one of the 
study conditions, making it impossible to separate how much of the observed effect was 
due to the intervention and how much was due to the factor. 

The design of a study is the method by which intervention and comparison groups were assigned. 
A domain is a group of closely related outcomes. 

The effect size is a measure of the magnitude of an effect. The WWC uses a standardized 
measure to facilitate comparisons across studies and outcomes. 

A study is eligible for review and inclusion in this report if it falls within the scope of the 
review protocol and uses either an experimental or matched comparison group design. 

A demonstration that the analysis sample groups are similar on observed characteristics 
defined in the review area protocol. 

An indication of how much evidence supports the findings. The criteria for the extent 
of evidence levels are given in the WWC Rating Criteria on p.18. 

Along a percentile distribution of individuals, the improvement index represents the gain 
or loss of the average individual due to the intervention. As the average individual starts at 
the 50th percentile, the measure ranges from -50 to +50. 

An educational program, product, practice, or policy aimed at improving student outcomes. 

A summary of the findings of the highest-quality research on a given program, product, 
practice, or policy in education. The WWC searches for all research studies on an interven- 
tion, reviews each against design standards, and summarizes the findings of those that 
meet WWC design standards. 

When a study includes multiple outcomes or comparison groups, the WWC will adjust 
the statistical significance to account for the multiple comparisons, if necessary. 

A quasi-experimental design (QED) is a research design in which study participants are 
assigned to intervention and comparison groups through a process that is not random. 

A randomized controlled trial (RCT) is an experiment in which eligible study participants are 
randomly assigned to intervention and comparison groups. 

The WWC rates the effects of an intervention in each domain based on the quality of the 
research design and the magnitude, statistical significance, and consistency in findings. The 
criteria for the ratings of effectiveness are given in the WWC Rating Criteria on p. 18. 

A research approach in which an outcome variable is measured repeatedly within and 
across different conditions that are defined by the presence or absence of an intervention. 
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Glossary of Terms 


Standard deviation The standard deviation of a measure shows how much variation exists across observations 
in the sample. A low standard deviation indicates that the observations in the sample tend 
to be very close to the mean; a high standard deviation indicates that the observations in 
the sample tend to be spread out over a large range of values. 

Statistical significance Statistical significance is the probability that the difference between groups is a result of 

chance rather than a real difference between the groups. The WWC labels a finding statistically 
significant if the likelihood that the difference is due to chance is less than 5% (p < .05). 


Substantively important a substantively important finding is one that has an effect size of 0.25 or greater, regardless 

of statistical significance. 

Systematic review a review of existing literature on a topic that is identified and reviewed using explicit meth- 
ods. A WWC systematic review has five steps: 1) developing a review protocol; 2) searching 
the literature; 3) reviewing studies, including screening studies for eligibility, reviewing the 
methodological quality of each study, and reporting on high quality studies and their find- 
ings; 4) combining findings within and across studies; and, 5) summarizing the review. 


Please see the WWC Procedures and Standards Handbook (version 3.0) for additional details. 
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Intervention 

Report 



Practice 

Guide 



Quick 

Review 


Single Study 
Review 



An intervention report summarizes the findings of high-quality research on a given program, practice, or policy in 
education. The WWC searches for all research studies on an intervention, reviews each against evidence standards, 
and summarizes the findings of those that meet standards. 


This intervention report was prepared for the WWC by Development Services Group under contract ED-IES-12-C-0084. 
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